System and method for speaker role determination and scrubbing identifying information

ABSTRACT

Methods for speaker role determination and scrubbing identifying information are performed by systems and devices. In speaker role determination, data from an audio or text file is divided into respective portions related to speaking parties. Characteristics classifying the portions of the data for speaking party roles are identified in the portions to generate data sets from the portions corresponding to the speaking party roles and to assign speaking party roles for the data sets. For scrubbing identifying information in data, audio data for speaking parties is processed using speech recognition to generate a text-based representation. Text associated with identifying information is determined based on a set of key words/phrases, and a portion of the text-based representation that includes a part of the text is identified. A segment of audio data that corresponds to the identified portion is replaced with different audio data, and the portion is replaced with different text.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The instant application is related to U.S. patent application Ser. No.16/397,745 entitled “SYSTEM AND METHOD FOR SPEAKER ROLE DETERMINATIONAND SCRUBBING IDENTIFYING INFORMATION,” and filed on the same dateherewith, the entirety of which is incorporated by reference herein.

BACKGROUND

Customers and other users may interact with support staff for providersof products and services. Interactions between the users and supportstaff may be performed via voice calls such as by telephone, voice overInternet protocol (VoIP), etc. The providers and/or the support staffmay record/save these interactions for user files, and also fordetermining whether the support staff meets desired metrics and criteriafor handling user and customer interactions. Audio data may be separatedinto segments that correspond to different participants of theinteractions through speaker diarization, however the participants'roles in the conversation remain unknown. Additionally, identifyinginformation for users and customers may be provided during interactions.The recording of such identifying information presents risks to privacyand data security.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Methods for speaker role determination and scrubbing identifyinginformation are performed by systems and devices. In one aspect,identification of speaker roles in conversations is performed for audioand/or text data. In speaker role determinations, data from audio filesand/or text files is divided into portions related to different speakingparties. In embodiments, characteristics that classify the portions ofthe data according to speaking party roles are identified in theportions of the data. The portions are aggregated to generate data setsthat correspond to each of the speaking party roles. A speaking partyrole for each of the data sets may be assigned based on the classifyingcharacteristics used to generate the data sets. In another aspect,scrubbing of identifying information for users and customers may also beperformed for audio and/or text data. For scrubbing identifyinginformation from data, audio data for speaking parties is processedusing automatic speech recognition to generate a text-basedrepresentation of the audio data. Text associated with identifyinginformation is determined based on a set of key words, key phrases,and/or patterns. A portion of the text-based representation thatincludes a part of the text associated with the identifying informationis identified for scrubbing. A segment of the audio data thatcorresponds to the identified portion is replaced with different,replacement audio data, and the portion of the text-based representationthat includes the part of the text associated with the identifyinginformation is replaced with different, replacement text.

Further features and advantages, as well as the structure and operationof various examples, are described in detail below with reference to theaccompanying drawings. It is noted that the ideas and techniques are notlimited to the specific examples described herein. Such examples arepresented herein for illustrative purposes only. Additional exampleswill be apparent to persons skilled in the relevant art(s) based on theteachings contained herein.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

The accompanying drawings, which are incorporated herein and form a partof the specification, illustrate embodiments of the present applicationand, together with the description, further serve to explain theprinciples of the embodiments and to enable a person skilled in thepertinent art to make and use the embodiments.

FIG. 1 shows a block diagram of a networked system for determiningspeaker roles and scrubbing identifying information, according to anexample embodiment.

FIG. 2 shows a block diagram of a computing system configured fordetermining speaker roles and scrubbing identifying information,according to an example embodiment.

FIG. 3 shows a flowchart for determining speaker roles, in accordancewith an example embodiment.

FIG. 4 shows a flowchart for determining speaker roles, in accordancewith an example embodiment.

FIG. 5 shows a flowchart for determining speaker roles, in accordancewith an example embodiment.

FIG. 6 shows a flowchart for maintaining and/or processing ML (machinelearning) model training data, in accordance with an example embodiment.

FIG. 7 shows a flowchart for scrubbing identifying information fromaudio and/or text data, according to an example embodiment.

FIG. 8 shows a flowchart for scrubbing identifying information fromaudio and/or text data using a range or boundary, in accordance with anexample embodiment.

FIG. 9 shows a flow diagram of the scrubbing of identifying informationin audio/text files as in FIGS. 7-8, in accordance with exampleembodiments.

FIG. 10 shows a block diagram of an example mobile device that may beused to implement various example embodiments.

FIG. 11 shows a block diagram of an example computing device that may beused to implement embodiments.

The features and advantages of embodiments will become more apparentfrom the detailed description set forth below when taken in conjunctionwith the drawings, in which like reference characters identifycorresponding elements throughout. In the drawings, like referencenumbers generally indicate identical, functionally similar, and/orstructurally similar elements. The drawing in which an element firstappears is indicated by the leftmost digit(s) in the correspondingreference number.

DETAILED DESCRIPTION I. Introduction

The following detailed description discloses numerous embodiments. Thescope of the present patent application is not limited to the disclosedembodiments, but also encompasses combinations of the disclosedembodiments, as well as modifications to the disclosed embodiments.

References in the specification to “one embodiment,” “an embodiment,”“an example embodiment,” etc., indicate that the embodiment describedmay include a particular feature, structure, or characteristic, butevery embodiment may not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to effect such feature, structure, or characteristicin connection with other embodiments whether or not explicitlydescribed.

In the discussion, unless otherwise stated, adjectives such as“substantially,” “approximately,” and “about” modifying a condition orrelationship characteristic of a feature or features of an embodiment ofthe disclosure, are understood to mean that the condition orcharacteristic is defined to be within tolerances that are acceptablefor operation of the embodiment for an application for which it isintended.

Furthermore, it should be understood that spatial descriptions (e.g.,“above,” “below,” “up,” “left,” “right,” “down,” “top,” “bottom,”“vertical,” “horizontal,” etc.) used herein are for purposes ofillustration only, and that practical implementations of the structuresand drawings described herein can be spatially arranged in anyorientation or manner. Additionally, the drawings may not be provided toscale, and orientations or organization of elements of the drawings mayvary in embodiments.

Numerous exemplary embodiments are described as follows. It is notedthat any section/subsection headings provided herein are not intended tobe limiting. Embodiments are described throughout this document, and anytype of embodiment may be included under any section/subsection.Furthermore, embodiments disclosed in any section/subsection may becombined with any other embodiments described in the samesection/subsection and/or a different section/subsection in any manner.

Section II below describes example embodiments. Specifically, SectionII.A describes embodiments for determining speaking roles, and SectionII.B describes embodiments for scrubbing identifying information.Section III below describes example mobile device and computing deviceembodiments that may be used to implement features of the embodimentsdescribed herein. Section IV below describes additional examples andadvantages, and Section V provides some concluding remarks.

II. Example Embodiments for Determining Speaking Roles and ScrubbingIdentifying Information

Methods for speaker role determination and scrubbing identifyinginformation are performed by systems and devices. Identifying speakerroles in conversations may be performed for audio and/or text data.Embodiments herein provide for analyzing telephone calls or recordingsthereof, e.g., conducted by call centers and/or produced in otherenvironments, to produce conversation metrics that can be used toevaluate the performance of call participants, such as support staffagents, and/or can be used for other purposes. For example, one metricmay indicate whether an agent speaks versus listens more to thecustomer. However, to properly determine and analyze conversationmetrics, the speaking roles of participants for the interaction must bedetermined, and many current implementations do not utilize separatechannels for different speakers in such interactions.

In speaker role determinations according to embodiments, data from audiofiles and/or text files is divided into portions related to thedifferent speaking parties. A speaker diarization technique or service(“diarization” hereinafter) may be utilized to help segment telephonecalls into speaker segments (i.e., which speaker speaks and when), butdiarization does not identify speaker roles. Accordingly, there is aneed to know which speaker is the agent and which speaker is thecustomer. According to embodiments herein, characteristics that classifythe portions of the data according to speaking party roles areidentified in the portions of the data. The data, as noted, may be audioor text data. In embodiments, text data may be classified/identifiedbased on textual patterns, and audio data may be classified/identifiedbased on comparisons of the audio data to known speaker audio samples.In embodiments for textual information utilization in text data, thetextual patterns may be based on words, phrases, speech patterns,timing, etc., that are most typically associated with a given speakingparty role. For example, a support staff agent may use questionwords/phrases (e.g., “what,” “when,” “how,” “may I help you,” and/or thelike), may address the other speaker by name with a title (e.g., “Mrs.Smith,” “Mr. Smith,” etc.), may refer to the other speaker as “you”instead of “I” and use possessive words such as “your” instead of“mine,” may refer to new issue numbers, may be the first speaker in theconversation, and/or the like.

The identified portions are aggregated to generate data sets thatcorrespond to each speaking party role. A speaking party role for eachof the data sets may be assigned based on the classifyingcharacteristics used to generate the data sets. In embodiments, thedetermination of speaking party roles may be made utilizing machinelearning (ML) algorithms that may be trained based on storedconversations and editing distances, as described in further detailherein. For instance, a binary classification model may be used fortwo-speaker conversations to predict whether the first speaker dataset/cluster is the agent or not. If the classifier returns a ‘false’result, then the first speaker data set/cluster is the customer and thesecond speaker data set/cluster is the agent; a ‘true’ result indicatesthe first speaker data set/cluster is the agent and the second speakerdata set/cluster is the customer.

Accordingly, one or both of the following approaches may be used todetermine and label speaker roles: (a) a text-based approach to assignspeaker role labels by looking at the words in each diarizationsubset/cluster, or (b) an acoustic based approach for audio data inwhich agents enroll their voices to build a biometric signature.

Scrubbing identifier information for users and customers may also beperformed for audio and/or text data. As noted above, conversationrecordings may be used to determine conversation metrics (e.g., forsupport staff agent performance) and/or for developing models via ML todetermine speaking roles in conversations. Transcripts, audiorecordings, and/or phone calls may be stored, indexed, and processed forcall centers, VoIP conversations, other types of transcripts, etc. Theremoval of end user identifying information (EUII), or “identifyinginformation” hereinafter, such as email addresses, phone numbers, creditcard numbers, social security numbers, account numbers, etc., from audiorecordings and transcriptions is important to maintain user privacy anddata security. However, these kinds of identifying information aredifficult to correctly identify themselves in audio recordings. Forexample, it may take multiple turns between speakers to communicate andconfirm identifying information in conversations, and speech recognizersmake many mistakes in their determinations for such conversations.

For instance, in a scenario in which a speaker tells her password, e.g.,“Pi10t” which includes alphanumeric characters and specific punctuationthat need to be pronounced and specified individually, to her spouse ora support staff agent over a voice call, it may take more than one,perhaps several, rounds of the conversation with speech from each party,including confirmation, to successfully deliver the information. In thisscenario, the speaker may say the password which, after transcribing,may result in: “pilot capital ‘p’ eye one zero tee.” It is also possiblethat the ASR (automatic speech recognition) may transcribe ‘1’ as “won”instead of “one,” or that ‘t’ may be transcribed as “tea,” or that“capital ‘p’” may be transcribed incorrectly, and such mistakes can leadto misidentification of the password. Moreover, if the speaker mustrepeat some or all of the password to ensure that the listening partyhas the password correct, the password will exist is in whole or in partat several locations in the text. With only ASR transcriptions and/ordiarization, the identifying information is very difficult to correctlyidentify from audio recordings.

Additionally, the removal of too much data from a conversation in orderto be sure the identifying information is caught risks the removal ofvaluable data and context used for metrics, as noted above. That is,preserving as much of the conversation and its context, i.e., what isgoing on in the conversation, while scrubbing off personal data so thata customer cannot be identified through the conversation data, providesfor useful leftover data that remains valuable for generating insights.Embodiments herein provide for contextual algorithms based on anunderstanding of ASR limitations to remove requisite portions ofidentifying information based on the surrounding context to ensureprivacy/data security while not removing contextually useful portions ofthe conversation.

In embodiments, for scrubbing identifier information from data, audiodata for speaking parties may be processed using automatic speechrecognition to generate a text-based representation of the audio data.Text associated with identifying information may be determined based ona set of key words and/or key phrases that are predefined orpredetermined and stored. In this description, the term key phrases mayalso encompass a key word or key words. A portion of the text-basedrepresentation that includes a part of the text associated that is withthe identifying information is identified for scrubbing. In embodiments,this portion may be larger than just the identifying information, andmay include additional context information, e.g., the word “email,” thephrase “dot com,” a length of numbers/alphanumeric character such as a16-digit number for a credit card, a 7- or 10-digit telephone number,etc., while other words that may be misconstrued or confused by ASR,e.g., “at” may or may not represent ‘@’, may be ignored as contextualkey phrases but may be used as context when combined with “email” and/or“dot com.” A segment of the audio data that corresponds to theidentified portion may be replaced with different audio data, such asbut not limited to, silence, a constant tone, etc., and the portion ofthe text-based representation that includes the part of the textassociated with the identifying information is replaced with differenttext, including but without limitation, blank characters, asterisks(‘*’) hyphens (‘-’), descriptors, and/or the like.

In embodiments, a length (e.g., a number of characters) and/or acontextual boundary may be used to limit scrubbing. When a length ornumber of characters is used to scrub identifying information, an upperlimit may be placed on the length/number of characters, which may bebased on the type of identifying information and/or the context, e.g.,few email addresses contain 40 characters or more, etc. For aboundary-limited scrubbing, the context of the text may be utilized todetermine how much of the content is to be scrubbed, e.g., if an emailaddress is identified, “dot com” may be scrubbed along with precedingcharacters back to a phrase such as “my email is,” “email,” and/or thelike. In embodiments, a meta-description of the scrubbed material may beused as a replacement for scrubbed text, e.g., “EMAIL-ADDRESS,”“CREDIT-CARD-NUMBER,” other descriptors, etc., for the benefit of laterreview and/or analytical algorithms.

Accordingly, instances of identifying information may be removed fromone or more of a text representation of audio data for a conversationand/or the audio data itself while maintaining the maximum amount ofdata and context for the conversation.

These and further embodiments will be described in further detail below,and in the Sections and Subsections that follow.

Systems, devices, and apparatuses may be configured in various ways toperform their functions for determining speaker roles in conversationsand/or for scrubbing identifying information from audio/text files forconversations. For instance, FIG. 1 is a block diagram of a networkedsystem 100, according to embodiments. System 100 is configured to enabledeterminations of speaking roles in text/audio conversations, and toscrub identifying information from text/audio conversations, accordingto embodiments. As shown in FIG. 1, system 100 includes a host device102, a client device(s) 104, and a machine learning (ML) host 118. Inembodiments, host device 102, ML host 118, and client device(s) 104 maycommunicate with each other over a network 110. It should be noted thatvarious numbers of host devices, client devices, and/or ML hosts may bepresent in various embodiments. Additionally, any combination of thecomponents illustrated in FIG. 1 may be present in system 100, accordingto embodiments.

As noted above, host device 102, client device(s) 104, and ML host 118are communicatively coupled via network 110. Network 110 may compriseany type of communication links that connect computing devices andservers such as, but not limited to, the Internet, wired or wirelessnetworks and portions thereof, point-to-point connections, local areanetworks, enterprise networks, and/or the like. In some embodiments,e.g., for legacy recordings, data may also be transferred, in additionto or in lieu of, using a network, on physical storage media, betweenclient device(s) 104, ML host 118, and/or host device 102.

ML host 118 may comprise one or more server computers or computingdevices, which may include one or more distributed or “cloud-based”servers. In embodiments, ML host 118 may be associated with, or may be apart of, a cloud-based service platform such as Microsoft® Azure® fromMicrosoft Corporation of Redmond, Wash. Host devices such as host device102 and/or client devices such as client device(s) 104 may be configuredto provide data files, including audio and text files, associated withconversations or speech to ML host 118 via network 110. ML host 118 maybe configured to train/re-train/generate models, algorithms, etc., fordetermining speaker roles in conversations and/or for scrubbingidentifying information from audio/text files for conversations. Inembodiments, neural networks may be implemented by ML host 118.

It should be noted that as described herein, embodiments of ML host 118are applicable to any type of system where a system for machine learningcommunicates with client devices over a network. One example noted aboveis where ML host 118 is a “cloud” implementation, application, orservice in a network architecture/platform. A cloud platform may includea networked set of computing resources, including servers, routers,etc., that are configurable, shareable, provide data security, and areaccessible over a network such as the Internet. Cloudapplications/services such as for machine learning may run on thesecomputing resources, often atop operating systems that run on theresources, for entities that access the applications/services over thenetwork. A cloud platform may support multi-tenancy, where cloudplatform-based software services multiple tenants, with each tenantincluding one or more users who share common access to software servicesof the cloud platform. Furthermore, a cloud platform may supporthypervisors implemented as hardware, software, and/or firmware that runvirtual machines (emulated computer systems, including operatingsystems) for tenants. A hypervisor presents a virtual operating platformfor tenants.

Client device 102 may be any type or combination of computing device orcomputing system, including a terminal, a personal computer, a laptopcomputer, a tablet device, a smart phone, a personal digital assistant,a telephone, and/or the like, including internal/external storagedevices, that may be utilized to record conversations, e.g.,conversations between speaking parties. Conversations may take place viaa telephone component of client device 102, though VoIP, etc., and maybe stored/recorded as audio files and/or text files. In embodiments,client device 102 may be used by various types of users, such as anadministrator, support staff agents, customers, clients, and/or the likethat participate in interactions with audio and/or text.

System 100 may additionally include a communication device 112communicatively coupled to client device 102. For instance, inembodiments for which client device 102 does not specifically includehardware/software to interact in a conversation with a remote entity,e.g., a technical support call from a client to a support staff agent,communication device 112 enables such conversations. Accordingly,communication device 112 may be any type of telephone, smart phone, etc.In some embodiments, client device(s) 104 may comprise a server orstorage device, e.g., as noted below for system 100, by whichconversations are recorded, transcribed, and/or stored.

Host device 102 may comprise one or more server computers or computingdevices, which may include one or more distributed or “cloud-based”servers, as described above. Host device 102 may include a roles manager108 that is configured to determine speaking roles for conversations, asdescribed herein, and may include a scrubbing manager 116 that isconfigured to scrub identifying information from conversations, asdescribed herein. While not shown for brevity and illustrative clarity,either or both of roles manager 108 and scrubbing manager 116 may beincluded in client device(s) 104.

System 100 may include a storage 106, that may be a stand-alone storagesystem (as shown), and/or may be internally or externally associatedwith host device 102. In embodiments, storage 106 may be communicativelycoupled to network 110. That is, storage 106 may be any type of storagedevice or array of devices, and while shown as being communicativelycoupled to host device 102, may be networked storage that is accessiblevia network 110. Additional instances of storage 106 may be included inaddition to, or in lieu of, the embodiment shown. Storage 106 isconfigured to store different types of data, including but not limitedto, audio data files for conversations, transcriptions or text files ofsuch audio data files or of other conversations, any types of documents,media/multi-media files, ML models, and/or the like.

A host device such as host device 102 (or a client device such as clientdevice(s) 104) may be configured in various ways for determining speakerroles and/or for scrubbing identifying information. For instance,referring now to FIG. 2, a block diagram of a system 200 is shown fordetermining speaker roles in conversations and/or for scrubbingidentifying information from audio/text files for conversations,according to an example embodiment. System 200 may be an embodiment ofsystem 100 of FIG. 1. System 200 is described as follows.

System 200 includes a computing device 202, which may be an embodimentof host device 102 of FIG. 1 (or client device(s) 104), and may be anytype of server or computing device, as mentioned elsewhere herein, or asotherwise known. As shown in FIG. 2, system 200 and computing device 202include one or more of a processor (“processor”) 204, one or more of amemory and/or other physical storage device (“memory”) 206, one or morenetwork interfaces (“network interface”) 226, a roles manager 208 thatmay be an embodiment of roles manager 108 of FIG. 1, and a scrubbingmanager 216 that may be an embodiment of scrubbing manager 116 ofFIG. 1. System 200 may also include additional components (not shown forbrevity and illustrative clarity) including, but not limited to,components and subcomponents of other devices and/or systems herein, aswell as those described below with respect to FIGS. 10 and 11, such asan operating system, etc.

Processor 204 and memory 206 may respectively be any type of processorcircuit and memory that is described herein, and/or as would beunderstood by a person of skill in the relevant art(s) having thebenefit of this disclosure. Processor 204 and memory 206 may eachrespectively comprise one or more processors or memories, differenttypes of processors or memories (e.g., a cache for query processing),remote processors or memories, and/or distributed processors ormemories. Processor 204 may be a multi-core processor configured toexecute more than one processing thread concurrently. Processor 204comprises circuitry that is configured to execute computer programinstructions such as, but not limited to, embodiments of roles manager208 and/or scrubbing manager 216, which may be implemented as computerprogram instructions for determining speaker roles and/or for scrubbingidentifying information, etc., as described herein. Memory 206 mayinclude storage 114 of FIG. 1 in embodiments, and may be configured tostore such computer program instructions/code, as well as to store otherinformation and data described in this disclosure including, withoutlimitation, one or more public/private key pairs shown as key(s) 222,etc.

Network interface 226 may be any type or number of wired and/or wirelessnetwork adapter, modem, etc., configured to enable system 200 tocommunicate with other devices over a network, such as communicationsbetween system 200 and other devices utilized in a network as describedherein (e.g., client device 104, host device 102, ML host 118, and/orstorage 106) over a network such as network 110 as described above withrespect to FIG. 1.

System 200 may also include an automatic speech recognition (ASR) engine228 and one or more ML models as model(s) 230. ASR engine 228 isconfigured to convert speech data from an audio file into a textrepresentation thereof. Model(s) 230 may represent ML models generatedby ML host 118 of FIG. 1.

Roles manager 208 includes a plurality of components for performing thefunctions and operations described herein for determining speaker roles.For instance, roles manager 208 is configured to assign speaking partyroles for one or more speaking parties in a conversation. As notedabove, a conversation may comprise recorded audio data in a first fileand/or may comprise transcribed text data from audio data in a textfile. In embodiments, as shown in FIG. 2, system 200 may include ASRengine 228 which may be configured to generate such transcribed textdata.

As shown, roles manager 208 includes an apportioner 210, acharacteristic identifier 212, and a data aggregator 214. While shownseparately for illustrative clarity, in embodiments, one or more ofapportioner 210, characteristic identifier 212, and/or data aggregator214, may be combined together and/or as a part of other components ofsystem 200. In some embodiments, less than all of the components ofroles manager 208 illustrated in FIG. 2 may be included. In softwareimplementations, one or more components of roles manager 208 may bestored in memory 206 and are executed by processor 204.

Apportioner 210 may be configured to divide data from audio and/or textfiles into portions. Apportioner 210 may perform this division viadiarization. A diarization service (not shown) itself may only separatean entire conversation for two speaking parties, e.g., from a telephonecall or transcription/ASR text, into two speaker clusters comprised ofthe respective portions. However, without the ability to identify whichgroup of portions (i.e., which cluster) is specifically associated witheach speaking party, the diarization alone is not sufficient.Accordingly, a “speaker role” label is needed for each cluster toidentify the agent. In other words, in order to properly generatemetrics and applications, such as application programming interfaces(APIs), two labeled clusters (e.g., agent vs. customer) are required. Itshould also be noted that with two speaking parties, identifying andlabeling one speaking party implicitly identifies and labels the otherspeaking party. In embodiments, characteristic identifier 212 and dataaggregator 214 are configured to perform the remaining operations foridentifying and labeling speaking parties.

Characteristic identifier 212 may be configured to find speaking partyrole characteristics in each portion generated by apportioner 210 (e.g.,based on stored agent voice samples for audio, and/or based on textualpatterns for text), and to assign speaking party roles to data setscomprised of the portions. Data aggregator 214 may be configured togenerate the data sets. In embodiments, the data sets correspond todifferent speaking parties present in the data.

Accordingly, one or more of a text-based approach to assign speaker rolelabels by looking at the words in each diarization subset or an acousticbased approach for audio data in which agents enroll their voices tobuild a biometric signature may be used to determine and label speakerroles utilizing roles manager 208.

Roles manager 208, apportioner 210, characteristic identifier 212, anddata aggregator 214 are described in further detail below in SectionII.A.

Scrubbing manager 216 includes a plurality of components for performingthe functions and operations described herein for scrubbing identifyinginformation. For instance, scrubbing manager 216 is configured to scrubidentifying information for one or more speaking parties in aconversation. As shown, scrubbing manager 216 includes a speechrecognizer 218, a context determiner 220, a text identifier 222, and ascrubber 224. While shown separately for illustrative clarity, inembodiments, one or more of speech recognizer 218, context determiner220, text identifier 222, and/or scrubber 224, may be combined togetherand/or as a part of other components of system 200. In some embodiments,less than all of the components of scrubbing manager 216 illustrated inFIG. 2 may be included. In software implementations, one or morecomponents of scrubbing manager 216 may be stored in memory 206 and areexecuted by processor 204.

Speech recognizer 218 may be configured to receive audio data filesassociated with a speaking party(ies) and generate text-basedrepresentations of the audio data. In embodiments, as shown in FIG. 2,system 200 may include ASR engine 228 which may be configured togenerate such text-based representations of the audio data. ASR engine228 may comprise a portion of speech recognizer 218, or may be aseparate component that is accessed by speech recognizer 218. In someembodiments, a text file with text data may be received instead of anaudio data file.

Context determiner 220 may be configured to determine the contextassociated with portions of the text-based representation based on keyword/key phrase comparisons. For example, key phrases that indicate acontext associated with identifying information may be determined bycontext determiner 220. Text identifier 222 may be configured toidentify parts of the text, based on the context, that includeidentifying information of a customer or user. That is, within theportions of text that have a context indicative of identifyinginformation, only a specific part of those portions may be identifiedfor scrubbing in order to leave as much of the original text (that isnot identifying information) not scrubbed. Scrubber 224 may beconfigured to scrub data that relates to identifying information inaudio and/or text files.

Accordingly, instances of identifying information may be removed fromone or more of a text representation of audio data for a conversationand/or the audio data itself, while maintaining the maximum amount ofdata and context for the conversation, utilizing scrubbing manager 216.

Scrubbing manager 216, speech recognizer 218, context determiner 220,text identifier 222, and/or scrubber 224 are described in further detailbelow in Section II.B.

A. Additional Embodiments for Determining Speaking Roles

As noted above for FIGS. 1 and 2, embodiments herein provide fordeterminations of speaker roles in conversations. System 100 of FIG. 1and system 200 of FIG. 2 may each be configured to perform suchdeterminations. For instance, FIGS. 3, 4, and 5 will now be described.FIG. 3 shows a flowchart 300, FIG. 4 shows a flowchart 400, and FIG. 5shows a flowchart 500, each being for determinations of speaker roles inconversations, according to example embodiments. Role manager 208 mayoperate according to flowchart 300, flowchart 400, and/or flowchart 500in embodiments. Further structural and operational examples will beapparent to persons skilled in the relevant art(s) based on thefollowing descriptions. Flowchart 300, flowchart 400, and/or flowchart500 are described as follows with respect to system 100 of FIG. 1 andsystem 200 of FIG. 2.

Flowchart 300 begins at step 302. In step 302, a file that includes datarelated to one or more of a first speaking party or a second speakingparty is received. For example, apportioner 210 of FIG. 2 may beconfigured to receive the file. The file may include text- oraudio-based data of a conversation, e.g., a telephone call between acustomer and a support staff agent although embodiments herein are notso limited. The file may be received via network 110 from clientdevice(s) 104 as in FIG. 1, from storage 106 as in FIG. 1, or frommemory 206 shown in FIG. 2. In embodiments, apportioner 210 may receivea text-based file from a storage device subsequent to transcriptiontherefor by ASR engine 228, or the like.

In step 304, the data is divided into portions based on one or morecharacteristics of the data. For example, apportioner 210 may beconfigured to divide the data into portions. As noted above, such adivision may be performed by diarization techniques, and may be based atleast on characteristics of the data. In embodiments, the datacharacteristics utilized in step 304 may be specific to text- oraudio-based data, as described in further detail below. In one example,audio data may be analyzed to determine breaks or pauses in speechcomprising the audio data that are within a specified threshold value,phrases recognized as questions or other types of phrases, a break orpause in speech that exceeds the specified threshold, different voicesspeaking, and/or the like.

In step 306, classifying characteristics of speaking party roles in eachof the portions are identified. For instance, characteristic identifier212 may be configured to identify classifying characteristics ofspeaking party roles in each of the portions. Classifyingcharacteristics may include, but are not limited to, textual patternsfor text representations of audio data, audio samples for knownspeakers, etc.

Referring also now to FIG. 4, flowchart 400 may be performed forembodiments in which the data from the file of flowchart 300 is textdata. Flowchart 400 is described in view of step 304 and step 306 offlowchart 300. Flowchart 400 begins at step 402.

In step 402, text data is generated, via ASR, as a text representationof the audio data. For example, speech recognizer 218 and/or ASR engine228 of FIG. 2 may be configured to generate text representations ofaudio data for conversations between speaking parties. Audio files maybe stored in a non-volatile memory, e.g., memory 206 of FIG. 2, storage106 of FIG. 1, etc., upon receipt from client device(s) 104 andretrieved for performance of step 402, or may be temporarily stored involatile memory such as a RAM of memory 206 for real-time, or nearreal-time, transcription. In embodiments, step 402 may be performed atany time prior to step 304 of flowchart 300.

In step 404, one or more of the characteristics of the data isidentified as a pattern of speech in the text representation. Forexample, apportioner 210, or another component of system 200, may beconfigured to identify characteristics of the text representation thatinclude one or more patterns of speech. In an embodiment, a pattern ofspeech may include blocks of contiguous speech having breaks or pauseswithin a specified threshold value, phrases recognized as questions orother types of phrases, a break or pause in speech that exceeds thespecified threshold, and/or the like. In embodiments, step 404 may beperformed at any time prior to step 304 of flowchart 300.

In step 406, the data is divided into portions based on at least one ofa timing or the pattern of speech in the text representation. Forexample, as noted above apportioner 210 may be configured to divide thedata into portions based on one more patterns of speech. That is, speechin the text representations may be divided into portions according toblocks of contiguous speech, types of phrases, breaks or pauses inspeech, etc., where each determined pattern indicator may designate adivision. Step 406 may be an embodiment of step 304.

In step 408, one or more of the classifying characteristics of thespeaking party roles is identified based on textual patterns from thetext representation. For instance, characteristic identifier 212 may beconfigured to identify classifying characteristics of speaker roles intext based on textual patterns within the text or portions thereof. Inembodiments, textual patterns for text data may be based on words,phrases, speech patterns, timing, etc., that are most typicallyassociated with a given speaking party role, including but withoutlimitation, question words/phrases, addresses to the other speaker byname with a title, references to the other speaker as “you” instead of“I” and the use of possessive words such as “your” instead of “mine,”references to new issue numbers, a greeting as the first speaker in theconversation, and/or the like. Textual patterns may be stored in amemory, e.g., memory 206 of FIG. 2, storage 106 of FIG. 1, etc., andcompared against the text data for identifications/classifications. Step408 may be an embodiment of step 306 of flowchart 300.

Based on the results of flowchart 400, flowchart 300 may resume at step308.

Referring now to FIG. 5, flowchart 500 may be performed for embodimentsin which the data from the file of flowchart 300 is audio data.Flowchart 500 is described in view of steps 304 and 306 of flowchart300. Flowchart 500 begins at step 502.

In step 502, at least one stored audio sample for a speaker is accessedin a memory. For example, apportioner 210, or another component ofsystem 200, may be configured to access and/or retrieve (as part of theaccess) stored audio samples from a storage, such as a memory (e.g.,memory 206) or other storage device (e.g., storage 106) described hereinand provide the audio sample(s) to apportioner 210, or another componentof system 200, for performance of step 504 described below. In otherembodiments, the audio sample(s) may be accessed by processor 204 forperformance of step 504. In one example, a speaking party, e.g., asupport staff agent, may pre-enroll with his/her voice to create a‘voice font’ that is acoustically based. This voice font may be storedand indexed according to the name of the speaking party. Enrollment mayinclude recording the voice of the speaking party for a given period oftime, e.g., one minute. In other embodiments, an agent may adopt or beotherwise assigned to, or identified with, sample audio speaker clustersin the system. In embodiments, step 502 may be performed at any timeprior to step 304 of flowchart 300.

In step 504, one of the portions is analyzed against at least one storedaudio sample to identify the classifying characteristics of the speakingparty roles. For instance, characteristic identifier 212 may beconfigured to identify the classifying characteristics in step 504 basedon an analysis of a portion (e.g., via division in step 304) against theaudio sample(s) retrieved/accessed in step 502. Accordingly, for speakerrole identification in audio data, portions may be determined as havinga higher speaker identification score with respect to a ‘voice font’, ifthe agent name is known, or may be compared against all different audiosamples of agents enrolled if the agent name is unavailable. Step 506may be an embodiment of step 306.

It should be noted that the described embodiments include theconcurrent, partially concurrent, and/or serial performance of one ormore steps of flowchart 400 and flowchart 500. In such embodiments, aresult from either, or both, of flowchart 400 and flowchart 500 may beused in furtherance of flowchart 300 for determinations of speaker rolesin conversations.

Flowchart 300 continues with step 308. In step 308, respective data setscorresponding to the one or more of the first speaking party and/or thesecond speaking party are generated, from the portions, based at leaston the classifying characteristics identified. For example, dataaggregator 214 may be configured to generate data sets for speakingparties from the portions. That is, the classifying characteristicsidentified in step 306 (and/or in flowchart 400 and/or flowchart 500)may be utilized by aggregator 214 to generate data sets comprisingportions with similar classifying characteristics.

In step 310, a speaking party role is assigned for one of the data sets,and/or one of the data sets is stored in a memory with an annotationidentifying the speaking party role associated with the one of the datasets. For instance, characteristic identifier 212 may be configured toassign a speaking party role for one of the data sets generated byaggregator 214 in step 308 above. In one example, a data set of audiodata portions determined to be associated with a specific agent may beassigned an “agent” role while another data set for the sameconversation may be assigned a “customer” or “user” role. Likewise, adata set of text data determined to be associated with a textualpatterns of speech used by an agent may be assigned an “agent” rolewhile another data set for the same conversation may be assigned a“customer” or “user” role. A data set may be stored in a storage, suchas but without limitation, memory 206 and/or storage 106 by rolesmanager 208 or a component thereof, in embodiments. Such stored datasets may be stored with annotations that identify the speaking partyrole associated with the data set. For instance, a data set generatedbased on classifying characteristics of an “agent” may be stored with anassociated annotation for later use according to the embodiments herein.

Embodiments herein also provide for maintaining and/or processing ML(machine learning) model training data for models which may be used toperform determinations of speaker roles in conversations and/orscrubbing of identifying information. For example, referring again tosystem 100 of FIG. 1 and system 200 of FIG. 2, computing device 202 insystem 200 may include one or more model(s) 230 that are trained,generated, and/or updated by ML host 118 of system 100. In suchembodiments, training data for ML models may be maintained and/orprocessed to identify, per speaking party, portions of a conversationbased on editing distances, as described in further detail below.

In FIG. 6, a flowchart 600 is shown for maintaining and/or processing MLmodel training data, in accordance with an example embodiment. ML host118 of system 100 may operate according to flowchart 600 in embodiments.Further structural and operational examples will be apparent to personsskilled in the relevant art(s) based on the following descriptions.Flowchart 600 is described as follows with respect to system 100 of FIG.1 and system 200 of FIG. 2. Flowchart 600 begins with step 602.

In step 602, a transcription file of audio data is received, thetranscription file identifying first speech portions for a first speakerand for a second speaker. For instance, ML host 118 may be configured toreceive a transcription file of audio data that identifies speechportions for a first speaker and for a second speaker. Transcriptionfiles may be received from client device(s) 104, from host device 102,and/or from storage 106, in embodiments, where the transcription filescomprise text that represents speech/conversations between one, two,three, or more, speaking parties. First speech portions for the firstspeaker and second speech portions for the second speaker may beprovided or additionally generated in step 602 as a concatenated stringof text. For example, speech portions for the first speaker may beconcatenated together, e.g., temporally ordered, and then concatenatedwith similarly arranged speech portions for the second speaker. Inembodiments, these first speech portions may be generated or verified tohave complete accuracy with respect to the audio data, e.g., viahuman-performed transcription.

As a non-limited example, consider the following transcription file of aconversation between two parties: a support agent and a customer.[Agent]: “Good Morning, Amy speaking, How may I help you today?”[Customer]: “I haven't received my refund you promised last week.”[Agent]: “One moment. Let me locate your record.”

In step 604, second speech portions of the audio data are generatedbased on at least one of automatic speech recognition or diarization.For example, a machine learning algorithm of ML host 118, such as astatistical decision algorithm, may be configured to drive ASR and/ordiarization to generate the second speech portions of the audio data,where the second speech portions may be generated as determinations ofthe speech as well as being identified with different speaking parties.In some cases, however, the ASR and/or the diarization results may notbe completely accurate.

Continuing with the example from step 602, consider the followingrepresentation of an ASR or diarization result. [Speaker 1]: “Morningany speaking how may I help you.” [Speaker 2]: “Today heaven received myrefund you promised last.” [Speaker 1]: “week one let me look for yourrecord.” As can be seen when compared to the accurate transcriptionabove, some words themselves may be incorrect, some words may bemissing, and some words may not be associated with the proper speakingparty. Thus, comparisons to the accurate transcription may be made,utilizing the idea of editing distances, to determine whether Speaker 1corresponds to the Support Agent or the Customer, and likewise, whetherSpeaker 2 corresponds to the Customer or the Support Agent.

In step 606, a first editing distance is determined between the firstspeech portions and a first arrangement of the second speech portions.For instance, ML host 118 and/or roles manager 208 may be configured todetermine the first editing distance. In embodiments, an editingdistance may be a number of edits applied to an arrangement of thesecond speech portions in order for it to match the first speechportions. In the example scenario set out above, where the first speechportions may be concatenated, the second speech portions may be likewiseconcatenated to form a single string, e.g., with all portions for onespeaker preceding all portions for the other speaker. For instance, theASR/diarization result provided in the example in step 604 may beconcatenated as “Morning any speaking how may I help you week one let melook for your record Today heaven received my refund you promised last.”In this example, the speech portion(s) for Speaker 1 precede the speechportion(s) for Speaker 2, and accordingly, such a first arrangement isanalyzed via editing distance against the concatenation described forthe example in step 602 to determine if Support Agent/Customercorresponds to Speaker 1/Speaker 2, respectively (although step 606 mayinstead determine the possibility Speaker 2/Speaker 1, in embodiments).

It may thus be determined in step 606 that a first number of edits, afirst editing distance, must be applied to the ASR/diarizationconcatenation (Speaker 1-Speaker 2), e.g., to replace incorrect words,missing words, out of order words, words improperly associated with aspeaking party, etc., in order for Speaker 1-Speaker 2 to match theknown Support Agent-Customer concatenated transcription.

In step 608, a second editing distance is determined between the firstspeech portions and a second arrangement of the second speech portions.For instance, ML host 118 and/or roles manager 208 may be configured todetermine the second editing distance. Continuing with the examplescenario set out above, the ASR/diarization result provided in theexample in step 604 may be concatenated as “Today heaven received myrefund you promised last Morning any speaking how may I help you weekone let me look for your record.” In this example, the speech portion(s)for Speaker 2 precede the speech portion(s) for Speaker 1, andaccordingly, such a second arrangement is analyzed via editing distanceagainst the concatenation described for the example in step 602 todetermine if Support Agent/Customer corresponds to Speaker 2/Speaker 2,respectively (although step 608 may instead determine the possibilitySpeaker 1/Speaker 2, in embodiments).

It may thus be determined in step 608 that a second number of edits mustbe applied to the ASR/diarization concatenation (Speaker 2-Speaker 1 inorder for Speaker 2-Speaker 1 to match the known Support Agent-Customerconcatenated transcription.

In step 610, each of the second speech portions is assigned to the firstspeaker or the second speaker based on the first editing distance andthe second editing distance. For example, continuing with the examplescenario above, ML host 118 may be configured to assign ones of thesecond speech portions to the first speaker or the second speaker basedon the editing distances from step 606 and step 608. In other words, iffewer edits are needed to match Speaker 1-Speaker 2 to SupportAgent-Customer than to match speaker 2-Speaker 1 to SupportAgent-Customer, then ML host 118 may assign the speaking role associatedwith the Support Agent to Speaker 1, and thus also assign the speakingrole associated with the Customer to Speaker 2. Conversely, if thesecond editing distance to associate Speaker 2-Speaker 1 with SupportAgent-Customer is lower than the first editing distance, then ML host118 may assign the speaking role associated with the Support Agent toSpeaker 2, and thus also assign the speaking role associated with theCustomer to Speaker 1.

For the above example scenario, the Speaker 1-Speaker 2 concatenationhas a lower editing distance with respect to Support Agent-Customerbecause the Speaker 1-Speaker 2 concatenation is more similar to theSupport Agent-Customer concatenation, thus Speaker 1 is assigned asSupport Agent and Speaker 2 is assigned as Customer.

In step 612, a statistical decision algorithm of the machine learningmodel is modified based on said assigning. For instance, ML host 118 maybe configured to modify (e.g., train, generate, update, etc.) its MLalgorithm based on the assignments made in step 610. That is, thealgorithm on which model(s) 230 are based may be trained, generated,and/or updated according to a progressing/maintenance of training datafor learning through editing distances determined against known accuratetranscriptions of audio data.

B. Additional Embodiments for Scrubbing Identifying Information

Systems, devices, and apparatuses may be configured in various ways toperform their functions for scrubbing identifying information from audioand/or text data, such as, but not limited to, data related toconversations between speaking parties. For example, the embodimentsherein provide for systems and methods to scrub identifying information,as described herein.

Such scrubbing may be performed at different times with respect to otherembodiments described herein. For instance, scrubbing may be performedprior to ML model training, generating, updating and/or after speakingrole determinations, as described in the subsection above.

Referring again to system 200 in FIG. 2, computing device 202 mayinclude scrubbing manager 216 configured scrub identifying informationfrom data. As noted above, scrubbing manager 216 may include speechrecognizer 218, context determiner 220, text identifier 222, andscrubber 224, one or more of which may be configured to scrubidentifying information from audio and/or text data.

In FIG. 7, a flowchart 700 is shown for scrubbing identifyinginformation from audio and/or text data, in accordance with an exampleembodiment. Scrubbing manager 216 may operate according to flowchart 700in embodiments. Further structural and operational examples will beapparent to persons skilled in the relevant art(s) based on thefollowing descriptions. Flowchart 700 is described as follows withrespect to system 100 of FIG. 1 and system 200 of FIG. 2. Flowchart 700begins with step 702.

In step 702, a file is received that includes audio data of one or morespeaking parties. For example, speech recognizer 218 of scrubbingmanager 216 in system 200 (FIG. 2) may be configured to receive audiodata files associated with a speaking party(ies). The audio data filesmay be received from client device(s) 104 of system 100 in FIG. 1, orfrom a memory or storage associated with system 200.

In step 704, a text-based representation of the audio data is generatedvia automatic speech recognition. For instance, speech recognizer 218may be configured to generate text-based representations of the audiodata. The text-based representations may be generated by speechrecognition, e.g., utilizing ASR 228 of system 200, as described above,e.g., ASR engine 228 may comprise a portion of speech recognizer 218, ormay be a separate component that is accessed by speech recognizer 218.

In step 706, text associated with identifying information in thetext-based representation is determined based on a set of key phrases.For example, context determiner 220 may be configured to determine acontext associated with portions of a text-based representation based onkey phrase comparisons. That is, as described herein, the text-basedrepresentation may include contextual information, e.g., the word“email,” the phrase “dot com,” a length of numbers/alphanumericcharacter such as a 16-digit number for a credit card, a 7- or 10-digittelephone number, etc., while other words that may be misconstrued orconfused by ASR, e.g., “at” may or may not represent ‘@’, and may beignored as contextual key phrases but may be used as context whencombined with “email,” “dot com,” and/or the like. Context determiner220 is configured to determine the context of the portions based on suchinformation, words, phrases, etc.

In step 708, a portion of the text-based representation that includes apart of the text is identified. For instance, text identifier 222 may beconfigured to identify parts of the text, based on the context, thatinclude identifying information of a customer or user. As an example,within the portions of the text-based representation (that have acontext indicative of identifying information), only a specific part ofthose portions may be identified for scrubbing in order to leave as muchof the original text (that is not identifying information) unchanged.

As described herein, a portion of the text-based representation thatincludes a part of the text associated that is with the identifyinginformation may be identified by text identifier for scrubbing. Inembodiments, the portion may be larger than just the identifyinginformation to be removed, and may include other information that isvaluable for understanding the entire conversation. Accordingly, onlythe part of the portion may be identified for scrubbing. Determining arange or boundary of the identified part to be scrubbed/replaced may beperformed, as described below, according to FIG. 8.

In step 710, a segment of the audio data that corresponds to the portionof the text-based representation is replaced with different audio data.For example, scrubber 224 may be configured to replace, i.e., scrub,data that relates to identifying information in audio files. Inembodiments, scrubber 224 may replace a segment of the audio datacorresponding to the portion in step 708 with audio data that is notidentifying. The replacement audio data may be silence, a tone, etc., asdescribed herein. Further details regarding step 710 are provided belowwith respect to FIG. 9.

In step 712, the portion of the text-based representation is replacedwith different text. For example, scrubber 224 may be configured toscrub data that relates to identifying information in text files. Inembodiments, scrubber 224 may replace the portion of the text-basedrepresentation from step 708 with text data that is not identifying. Thereplacement text data may be blank spaces, asterisks, etc., as describedherein. Further details regarding step 712 are provided below withrespect to FIG. 9.

As noted above, data/information scrubbing may be performed according toa range or boundary, or more specifically in embodiments, to an initialboundary and a final boundary that encompass the scope of information tobe scrubbed as a range of data to be replaced.

In step 714, modified audio data or a modified text-based representationis stored based on removal of the identifying information. For example,modified audio data and/or modified text-based representations may bestored in a storage, such as but without limitation, memory 206 and/orstorage 106 by roles manager 208 or a component thereof, in embodiments.Such stored data/representations may be stored with annotations thatidentify it as scrubbed.

In FIG. 8, a flowchart 800 is shown for scrubbing identifyinginformation from audio and/or text data using a range or boundary, inaccordance with an example embodiment. Scrubbing manager 216 may operateaccording to flowchart 800 in embodiments. Flowchart 800 may be anembodiment of flowchart 700. Further structural and operational exampleswill be apparent to persons skilled in the relevant art(s) based on thefollowing descriptions. Flowchart 800 is described as follows withrespect to system 200 of FIG. 2. Flowchart 800 begins with step 802.

In step 802, a context of at least one of the text or other textassociated with the text is determined. For example, context determiner220 of scrubbing manager 216 in system 200 (FIG. 2) may be configured todetermine one or more contexts of any section of the text determined instep 706 of flowchart 700. For example, context determiner 220 may beconfigured to determine a context associated with portions of atext-based representation based on key phrase comparisons, as well asother text surrounding the key phrases. That is, as described herein,the text-based representation may include additional contextualinformation, as well as key phrases, such as, but without limitation,“send it to my email,” “no, it's,” “can you please repeat your PIN?,”“my security phrase is,” “preferred contact number,” “yes, it's,” “canyou please verify your address?” and/or the like.

Step 802 may be performed prior to step 708 of flowchart 700 in FIG. 7.From step 802, flowchart 800 may continue to step 804, or to step 808,in various embodiments.

In step 804, a boundary of identifying information is identified basedon the context. For instance, context determiner 220 may be configuredto identify boundaries for scrubbing identifying information. A boundarymay be identified/determined based on context as described herein, e.g.,in step 802. A context utilized by context determiner 220 may be acharacteristic of the text that is at least one of: a character lengthof the text, a character type of the text, or a phrase included in theother text, and/or the like, as described herein. As an example, contextdeterminer 220 may determine in step 706 that an email address, one typeof text that is identifying information, is present. Contextualinformation may be then used by context determiner 220 to identifyboundaries, e.g., the phrase “dot com” may represent the final boundary,and a phrase “send it to my email,” or the like as described herein, mayrepresent the initial boundary.

In step 806, a range of characters for the scrubber code is determinedto replace the segment based on the context and the text and/or theboundary of identifying information. For example, context determiner 220may be configured to determine a range of characters for the scrubber,to replace the segment, based on the context, the boundary information,and/or the text. In an example scenario, a text representation of acustomer providing their email address (e.g.,FirstName23.LastName@emailhostprovider.com) may be “my email is firstname twenty three dot last name at email host provider dot com.” Havingrecognized that an email address is provided via the context and keyphrases, and having identified, e.g., “my email is” as the initialboundary and “dot com” as the final boundary, context determiner 220 maybegin at some character within the email address and expand a range ofcharacters until both boundaries are reached. In this example, but notby way of limitation, the in “dot com” may be the starting point toexpand the range: “.com” represents 4 characters to the final boundary,and “FirstName23.LastName@emailhostprovider” represents 38 characters tothe initial boundary, for a total of 42 characters in the determinedrange to be applied by scrubber 224.

In embodiments, step 710 of flowchart 700 in FIG. 7 may be performedbased on step 804 and/or step 806 of flowchart 800. Flowchart 800 mayend after step 806 and not proceed to step 808.

From step 802, flowchart 800 may proceed to step 808. In step 808, arange of characters for the scrubber code is determined to replace thesegment based on the context and the text and/or on a maximum,predefined range for a type of identifying information in the text. Forinstance, context determiner 220 may be configured to determine a rangeof characters for scrubber 224 based on the context and the text and/oron a maximum, predefined range for the type of identifying information.In an example, an email address may be scrubbed up to 40 characters (orsome other desired number of characters) back from the end of “.com”(where in the example email address above “Fi” of “FirstName23” wouldremain un-scrubbed for a 40 character range) or forward from thebeginning of the email address (where “om” of “.com” would remain). Inembodiments, scrubbing may be desired to an initial or final boundary,while in other embodiments, a predetermined range of characters may beidentified for scrubbing.

In embodiments, step 710 of flowchart 700 in FIG. 7 may be performedbased on step 808 of flowchart 800.

FIG. 9 shows a flow diagram 900 for scrubbing of identifying informationin audio/text files, in accordance with an example embodiment. Scrubbingmanager 216 may operate according to flow diagram 900 in embodiments.Flow diagram 900 may be an embodiment of flowchart 700 and/or flowchart800 described above. Further structural and operational examples will beapparent to persons skilled in the relevant art(s) based on thefollowing descriptions. Flow diagram 900 is described as follows withrespect to system 200 of FIG. 2. Flow diagram 900 begins with step 902.

In step 902, a text-based representation 908 is generated using ASR froma received audio file 906, as similarly described in step 702 and step704 of flowchart 700. As shown, and as described for step 706 offlowchart 700, identifying information, e.g., a customer email address,is determined in a portion of text of text-based representation 908based on key phrases: “name at mailhost dot com.” In this describedexample, contextual information is used to identify the portion of textshown for text-based representation 908 “ . . . my email is ‘name atmailhost dot com’ . . . ,” as in step 708 of flowchart 700. Additionalsteps described in flowchart 800 may also be performed.

In step 904, scrubbing of text-based representation 908 and/or receivedaudio file 906 are performed as described in flowchart 700 and flowchart800. For instance, scrubbing of text-based representation 908 and/orreceived audio file 906 may be performed based on boundaries and/orranges. As illustrated in this non-limiting example, a scrubbed audiofile 910 and a scrubbed text-representation 912 are generated byscrubber 224. That is, a segment 914 of audio file 906 is replaced witha constant tone (e.g., audio data that is not identifying of a customer)in scrubbed audio file 910 (as described in step 710 of flowchart 700),and the text determined to be identifying information, i.e., an emailaddress, in text-based representation 908 is replaced in scrubbedtext-representation 912 (as described in step 712 of flowchart 700) asplaceholder characters 916 shown as asterisks ‘*’ (e.g., text data thatis not identifying of a customer).

In embodiments, a descriptor corresponding to the type of identifyinginformation may be used as replacement text for scrubbed identifyinginformation. For example, a descriptor 918 shown as “EMAIL-ADDRESS”which provides an indication of the type of identifying information thatwas scrubbed.

The scrubbed versions, scrubbed audio file 910 and/scrubbedtext-representation 912, may be then be saved in place of the originalversions, received audio file 906 and/or text-based representation 908,respectively, to provide for protection of user data and privacy.

III. Example Mobile Device and Computing Device Embodiments

Embodiments described herein may be implemented in hardware, or hardwarecombined with software and/or firmware. For example, embodimentsdescribed herein may be implemented as computer programcode/instructions configured to be executed in one or more processorsand stored in a computer readable storage medium. Alternatively,embodiments described herein may be implemented as hardwarelogic/electrical circuitry.

As noted herein, the embodiments described, including without limitationsystem 100 of FIG. 1 and system 200 of FIG. 2, along with any componentsand/or subcomponents thereof, as well as any flowcharts/flow diagramsdescribed herein, including portions thereof, and/or further examplesdescribed herein, may be implemented in hardware, or hardware with anycombination of software and/or firmware, including being implemented ascomputer program code configured to be executed in one or moreprocessors and stored in a computer readable storage medium, or beingimplemented as hardware logic/electrical circuitry, such as beingimplemented together in a system-on-chip (SoC), a field programmablegate array (FPGA), or an application specific integrated circuit (ASIC).A SoC may include an integrated circuit chip that includes one or moreof a processor (e.g., a microcontroller, microprocessor, digital signalprocessor (DSP), etc.), memory, one or more communication interfaces,and/or further circuits and/or embedded firmware to perform itsfunctions.

Embodiments described herein may be implemented in one or more computingdevices similar to a mobile system and/or a computing device instationary or mobile computer embodiments, including one or morefeatures of mobile systems and/or computing devices described herein, aswell as alternative features. The descriptions of mobile systems andcomputing devices provided herein are provided for purposes ofillustration, and are not intended to be limiting. Embodiments may beimplemented in further types of computer systems, as would be known topersons skilled in the relevant art(s).

FIG. 10 is a block diagram of an exemplary mobile system 1000 thatincludes a mobile device 1002 that may implement embodiments describedherein. For example, mobile device 1002 may be used to implement anysystem, client, or device, or components/subcomponents thereof, in thepreceding sections. As shown in FIG. 10, mobile device 1002 includes avariety of optional hardware and software components. Any component inmobile device 1002 can communicate with any other component, althoughnot all connections are shown for ease of illustration. Mobile device1002 can be any of a variety of computing devices (e.g., cell phone,smart phone, handheld computer, Personal Digital Assistant (PDA), etc.)and can allow wireless two-way communications with one or more mobilecommunications networks 1004, such as a cellular or satellite network,or with a local area or wide area network.

Mobile device 1002 can include a controller or processor 1010 (e.g.,signal processor, microprocessor, ASIC, or other control and processinglogic circuitry) for performing such tasks as signal coding, dataprocessing, input/output processing, power control, and/or otherfunctions. An operating system 1012 can control the allocation and usageof the components of mobile device 1002 and provide support for one ormore application programs 1014 (also referred to as “applications” or“apps”). Application programs 1014 may include common mobile computingapplications (e.g., email applications, calendars, contact managers, webbrowsers, messaging applications) and any other computing applications(e.g., word processing applications, mapping applications, media playerapplications).

Mobile device 1002 can include memory 1020. Memory 1020 can includenon-removable memory 1022 and/or removable memory 1024. Non-removablememory 1022 can include RAM, ROM, flash memory, a hard disk, or otherwell-known memory devices or technologies. Removable memory 1024 caninclude flash memory or a Subscriber Identity Module (SIM) card, whichis well known in GSM communication systems, or other well-known memorydevices or technologies, such as “smart cards.” Memory 1020 can be usedfor storing data and/or code for running operating system 1012 andapplication programs 1014. Example data can include web pages, text,images, sound files, video data, or other data to be sent to and/orreceived from one or more network servers or other devices via one ormore wired or wireless networks. Memory 1020 can be used to store asubscriber identifier, such as an International Mobile SubscriberIdentity (IMSI), and an equipment identifier, such as an InternationalMobile Equipment Identifier (IMEI). Such identifiers can be transmittedto a network server to identify users and equipment.

A number of programs may be stored in memory 1020. These programsinclude operating system 1012, one or more application programs 1014,and other program modules and program data. Examples of such applicationprograms or program modules may include, for example, computer programlogic (e.g., computer program code or instructions) for implementing oneor more of system 100 of FIG. 1 and system 200 of FIG. 2, along with anycomponents and/or subcomponents thereof, as well as the flowcharts/flowdiagrams described herein, including portions thereof, and/or furtherexamples described herein.

Mobile device 1002 can support one or more input devices 1030, such as atouch screen 1032, a microphone 1034, a camera 1036, a physical keyboard1038 and/or a trackball 1040 and one or more output devices 1050, suchas a speaker 1052 and a display 1054. Other possible output devices (notshown) can include piezoelectric or other haptic output devices. Somedevices can serve more than one input/output function. For example,touch screen 1032 and display 1054 can be combined in a singleinput/output device. Input devices 1030 can include a Natural UserInterface (NUI).

One or more wireless modems 1060 can be coupled to antenna(s) (notshown) and can support two-way communications between processor 1010 andexternal devices, as is well understood in the art. Modem 1060 is showngenerically and can include a cellular modem 1066 for communicating withthe mobile communication network 1004 and/or other radio-based modems(e.g., Bluetooth 1064 and/or Wi-Fi 1062). At least one wireless modem1060 is typically configured for communication with one or more cellularnetworks, such as a GSM network for data and voice communications withina single cellular network, between cellular networks, or between themobile device and a public switched telephone network (PSTN).

Mobile device 1002 can further include at least one input/output port1080, a power supply 1082, a satellite navigation system receiver 1084,such as a Global Positioning System (GPS) receiver, an accelerometer1086, and/or a physical connector 1090, which can be a USB port, IEEE1394 (FireWire) port, and/or RS-232 port. The illustrated components ofmobile device 1002 are not required or all-inclusive, as any componentscan be deleted and other components can be added as would be recognizedby one skilled in the art.

In an embodiment, mobile device 1002 is configured to implement any ofthe above-described features of flowcharts herein. Computer programlogic for performing any of the operations, steps, and/or functionsdescribed herein may be stored in memory 1020 and executed by processor1010.

FIG. 11 depicts an exemplary implementation of a computing device 1100in which embodiments may be implemented. For example, embodimentsdescribed herein may be implemented in one or more computing devicessimilar to computing device 1100 in stationary or mobile computerembodiments, including one or more features of computing device 1100and/or alternative features. The description of computing device 1100provided herein is provided for purposes of illustration, and is notintended to be limiting. Embodiments may be implemented in further typesof computer systems and/or game consoles, etc., as would be known topersons skilled in the relevant art(s).

As shown in FIG. 11, computing device 1100 includes one or moreprocessors, referred to as processor circuit 1102, a system memory 1104,and a bus 1106 that couples various system components including systemmemory 1104 to processor circuit 1102. Processor circuit 1102 is anelectrical and/or optical circuit implemented in one or more physicalhardware electrical circuit device elements and/or integrated circuitdevices (semiconductor material chips or dies) as a central processingunit (CPU), a microcontroller, a microprocessor, and/or other physicalhardware processor circuit. Processor circuit 1102 may execute programcode stored in a computer readable medium, such as program code ofoperating system 1130, application programs 1132, other programs 1134,etc. Bus 1106 represents one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. System memory 1104 includes readonly memory (ROM) 1108 and random access memory (RAM) 1110. A basicinput/output system 1112 (BIOS) is stored in ROM 1108.

Computing device 1100 also has one or more of the following drives: ahard disk drive 1114 for reading from and writing to a hard disk, amagnetic disk drive 1116 for reading from or writing to a removablemagnetic disk 1118, and an optical disk drive 1120 for reading from orwriting to a removable optical disk 1122 such as a CD ROM, DVD ROM, orother optical media. Hard disk drive 1114, magnetic disk drive 1116, andoptical disk drive 1120 are connected to bus 1106 by a hard disk driveinterface 1124, a magnetic disk drive interface 1126, and an opticaldrive interface 1128, respectively. The drives and their associatedcomputer-readable media provide nonvolatile storage of computer-readableinstructions, data structures, program modules and other data for thecomputer. Although a hard disk, a removable magnetic disk and aremovable optical disk are described, other types of hardware-basedcomputer-readable storage media can be used to store data, such as flashmemory cards, digital video disks, RAMs, ROMs, and other hardwarestorage media.

A number of program modules may be stored on the hard disk, magneticdisk, optical disk, ROM, or RAM. These programs include operating system1130, one or more application programs 1132, other programs 1134, andprogram data 1136. Application programs 1132 or other programs 1134 mayinclude, for example, computer program logic (e.g., computer programcode or instructions) for implementing embodiments described herein,such as but not limited to, system 100 of FIG. 1 and system 200 of FIG.2, along with any components and/or subcomponents thereof, as well asthe flowcharts/flow diagrams described herein, including portionsthereof, and/or further examples described herein.

A user may enter commands and information into the computing device 1100through input devices such as keyboard 1138 and pointing device 1140.Other input devices (not shown) may include a microphone, joystick, gamepad, satellite dish, scanner, a touch screen and/or touch pad, a voicerecognition system to receive voice input, a gesture recognition systemto receive gesture input, or the like. These and other input devices areoften connected to processor circuit 1102 through a serial portinterface 1142 that is coupled to bus 1106, but may be connected byother interfaces, such as a parallel port, game port, or a universalserial bus (USB).

A display screen 1144 is also connected to bus 1106 via an interface,such as a video adapter 1146. Display screen 1144 may be external to, orincorporated in computing device 1100. Display screen 1144 may displayinformation, as well as being a user interface for receiving usercommands and/or other information (e.g., by touch, finger gestures,virtual keyboard, etc.). In addition to display screen 1144, computingdevice 1100 may include other peripheral output devices (not shown) suchas speakers and printers.

Computing device 1100 is connected to a network 1148 (e.g., theInternet) through an adaptor or network interface 1150, a modem 1152, orother means for establishing communications over the network. Modem1152, which may be internal or external, may be connected to bus 1106via serial port interface 1142, as shown in FIG. 11, or may be connectedto bus 1106 using another interface type, including a parallelinterface.

As used herein, the terms “computer program medium,” “computer-readablemedium,” “computer-readable storage medium,” and “computer-readablestorage device,” etc., are used to refer to physical hardware media.Examples of such physical hardware media include the hard diskassociated with hard disk drive 1114, removable magnetic disk 1118,removable optical disk 1122, other physical hardware media such as RAMs,ROMs, flash memory cards, digital video disks, zip disks, MEMs,nanotechnology-based storage devices, and further types ofphysical/tangible hardware storage media (including memory 1120 of FIG.11). Such computer-readable media and/or storage media are distinguishedfrom and non-overlapping with communication media and propagatingsignals (do not include communication media and propagating signals).Communication media embodies computer-readable instructions, datastructures, program modules or other data in a modulated data signalsuch as a carrier wave. The term “modulated data signal” means a signalthat has one or more of its characteristics set or changed in such amanner as to encode information in the signal. By way of example, andnot limitation, communication media includes wireless media such asacoustic, RF, infrared and other wireless media, as well as wired media.Embodiments are also directed to such communication media that areseparate and non-overlapping with embodiments directed tocomputer-readable storage media.

As noted above, computer programs and modules (including applicationprograms 1132 and other programs 1134) may be stored on the hard disk,magnetic disk, optical disk, ROM, RAM, or other hardware storage medium.Such computer programs may also be received via network interface 1150,serial port interface 1142, or any other interface type. Such computerprograms, when executed or loaded by an application, enable computingdevice 1100 to implement features of embodiments discussed herein.Accordingly, such computer programs represent controllers of thecomputing device 1100.

Embodiments are also directed to computer program products comprisingcomputer code or instructions stored on any computer-readable medium orcomputer-readable storage medium. Such computer program products includehard disk drives, optical disk drives, memory device packages, portablememory sticks, memory cards, and other types of physical storagehardware.

IV. Additional Example and Advantages

As described, systems and devices embodying the techniques herein may beconfigured and enabled in various ways to perform their respectivefunctions. In embodiments, one or more of the steps or operations of anyflowchart and/or flow diagram described herein may not be performed.Moreover, steps or operations in addition to or in lieu of those in anyflowchart and/or flow diagram described herein may be performed.Further, in examples, one or more operations of any flowchart and/orflow diagram described herein may be performed out of order, in analternate sequence, or partially (or completely) concurrently with eachother or with other operations.

It is contemplated herein that different languages, other than Englishas used in the descriptions of embodiments, may be present inembodiments without altering the underlying concepts of the describedembodiments. The described embodiments herein also provide for morerobust data and privacy protection, while still maintaining as muchpossible context/information for the determination of conversationalsupport metrics. Moreover, the described embodiments do not exist insoftware implementations for speaker role determination and scrubbingidentifying information. Conventional solutions lack the ability toimplement contextual information in conversation data with diarization,speaker role determination, data scrubbing, and ML models/algorithms,and are thus prone to errors.

The additional examples and embodiments described in this Section may beapplicable to examples disclosed in any other Section or subsection ofthis disclosure.

Some embodiments above provide for systems, devices, and methods fordetermining speaking roles and processing/maintaining training data forML models. For instance, a system is described herein. The system may beconfigured and enabled in various ways for determining speaking roles,as described herein. The system includes at least one processor and amemory that stores computer program instructions that are executable bythe processor(s). The computer program instructions include anapportioner, a characteristic identifier, and a data aggregator. Theapportioner is configured to receive a file that includes data relatedto a first speaking party and a second speaking party, and divide thedata into portions based on one or more characteristics of the data. Thecharacteristic identifier is configured to identify classifyingcharacteristics of speaking party roles in each of the portions. Thedata aggregator is configured to generate, from the portions, respectivedata sets corresponding to the first speaking party and the secondspeaking party based at least on the identified classifyingcharacteristics. The characteristic identifier is also configured toassign a speaking party role for at least one of the data sets.

In an embodiment of the system, the one or more characteristics of thedata comprise a pattern of speech in the data.

In an embodiment of the system, the data comprises audio data, and toidentify the classifying characteristics of the speaking party roles,the characteristic identifier is configured to analyze one of theportions against at least one stored audio sample.

In an embodiment, the system further includes a speech recognizerconfigured to generate text data as a text representation of the audiodata. In the embodiment, the characteristic identifier is configured toidentify one or more of the classifying characteristics based on textualpatterns from the text representation.

In an embodiment of the system, the characteristic identifier isconfigured to identify the classifying characteristics based on the atleast one stored audio sample and the textual patterns at leastpartially concurrently.

In an embodiment of the system, the data comprises text data derived viaautomatic speech recognition of audio data, and the one or moreclassifying characteristics comprise a textual pattern in the text data.

In an embodiment of the system, to identify the classifyingcharacteristics, the characteristic identifier is configured to analyzethe textual pattern against at least one text pattern set associatedwith one or more speaking party roles.

In an embodiment of the system, the characteristic identifier isconfigured to implement a speaking roles model that is based at least onthe one or more classifying characteristics and that is trained by amachine learning algorithm that utilizes editing distances betweentranscribed representations of audio data and representations based onof one or more of automatic speech recognition of the audio data ordiarization of the audio data.

A computer-implemented method is also described herein. Thecomputer-implemented method may be for determining speaking roles, asdescribed herein. The computer-implemented method includes dividing datarelated to one or more of a first speaking party or a second speakingparty into portions based on one or more characteristics of the data,and identifying classifying characteristics of speaking party roles ineach of the portions. The computer-implemented method also includesgenerating, from the portions, respective data sets corresponding to theone or more of the first speaking party or the second speaking partybased at least on the classifying characteristics identified, and one ofthe data sets in a memory with an annotation identifying a speakingparty role associated with the one of the data sets. In an embodiment,the computer-implemented method includes assigning a speaking party rolefor one of the data sets.

In an embodiment of the computer-implemented method, the one or morecharacteristics of the data comprise a pattern of speech in the data.

In an embodiment of the computer-implemented method, the data comprisesaudio data, and identifying the classifying characteristics of thespeaking party roles includes analyzing one of the portions against atleast one stored audio sample.

In an embodiment, the computer-implemented method includes generatingtext data as a text representation of the audio data, and identifyingone or more of the classifying characteristics is based on textualpatterns from the text representation.

In an embodiment, the computer-implemented method includes identifyingthe classifying characteristics based on the at least one stored audiosample and the textual patterns at least partially concurrently.

In an embodiment of the computer-implemented method, the data comprisestext data derived via automatic speech recognition of audio data, andthe one or more classifying characteristics comprise a textual patternin the text data.

In an embodiment of the computer-implemented method, identifyingclassifying characteristics includes analyzing the textual patternagainst at least one text pattern set associated with one or morespeaking party roles.

In an embodiment of the computer-implemented method, identifyingclassifying characteristics is performed utilizing a speaking rolesmodel that is based at least on the one or more classifyingcharacteristics and that is trained by a machine learning algorithm thatutilizes editing distances between transcribed representations of audiodata and representations based on of one or more of automatic speechrecognition of the audio data or diarization of the audio data.

In an embodiment of the computer-implemented method, the speaking rolesmodel comprises a statistical probability algorithm that indicates alikelihood of a given one of the portions being associated with one ofthe one or more of the first speaking party or the second speakingparty, and generating is based on the statistical probability algorithm.

A computer-readable storage medium having program instructions recordedthereon that, when executed by a processing device, perform a method formaintaining and/or processing ML (machine learning) model training data,is also described. The method includes receiving a transcription file ofaudio data, the transcription file identifying first speech portions fora first speaker and for a second speaker, and generating second speechportions of the audio data based on at least one of automatic speechrecognition or diarization. The method also includes determining a firstediting distance between the first speech portions and a firstarrangement of the second speech portions, and determining a secondediting distance between the first speech portions and a secondarrangement of the second speech portions. The method further includesassigning each of the second speech portions to the first speaker or thesecond speaker based on the first editing distance and the secondediting distance, and modifying a statistical decision algorithm of themachine learning model based on said assigning.

In an embodiment of the computer-readable storage medium, the firstediting distance includes a first number of edits applied to the firstarrangement of the second speech portions in order to match the firstspeech portions and the second editing distance includes a second numberof edits applied to the second arrangement of the second speech portionsin order to match the first speech portions.

In an embodiment of the computer-readable storage medium, in the method,generating is based on the automatic speech recognition and thediarization.

Some embodiments above provide for systems, devices, and methods forscrubbing identifying information from data, such as audio and/or textdata. For instance, a system is described herein. The system may beconfigured and enabled in various ways for scrubbing identifyinginformation, as described herein. The system includes at least oneprocessor and a memory that stores computer program instructions thatare executable by the processor(s). The computer program instructionsinclude a speech recognizer, a context determiner, a text identifier,and a scrubber. The speech recognizer is configured to receive a filethat includes audio data of one or more speaking parties, and generate atext-based representation of the audio data via automatic speechrecognition. The context determiner is configured to determine textassociated with identifying information in the text-based representationbased on a set of key phrases and/or key words. The text identifier isconfigured to identify a portion of the text-based representation thatincludes a part of the text, and the scrubber configured to replace asegment of the audio data that corresponds to the portion of thetext-based representation with different audio data.

In an embodiment of the system, the scrubber is configured to replacethe portion of the text-based representation with different text.

In an embodiment of the system, the context determiner is configured todetermine a context of at least one of the text or other text associatedwith the text, and determine a range of characters for the scrubber, toreplace the segment, based on the context and the text. In theembodiment, a context for the text comprises a characteristic of thetext from at least one of a character length of the text, a charactertype of the text, or a phrase included in the other text. In anembodiment, the context determiner is configured to identify a boundaryof identifying information based on the context, and determine the rangeof characters based on the boundary of identifying information. In anembodiment, the context determiner is configured to determine the rangeof characters based on a maximum, predefined range for a type ofidentifying information in the text.

In an embodiment of the system, the scrubber is configured to replacethe segment based on timing information of the generated text-basedrepresentation.

A computer-implemented method is also described herein. Thecomputer-implemented method may be for scrubbing identifyinginformation, as described herein. The computer-implemented methodincludes generating a text-based representation of audio data from afile via automatic speech recognition, the audio data including data ofone or more speaking parties, and determining text associated withidentifying information in the text-based representation based on a setof key phrases and/or key words. The computer-implemented method alsoincludes identifying a portion of the text-based representation thatincludes a part of the text, and replacing a segment of the audio datathat corresponds to the portion of the text-based representation withdifferent audio data. In an embodiment, the different audio dataprevents determination of the identifying information.

In an embodiment, the computer-implemented method includes replacing theportion of the text-based representation with different text thatprevents determination of the identifying information.

In an embodiment, the computer-implemented method includes determining acontext of at least one of the text or other text associated with thetext, and determining a range of characters to replace the segment basedon the context and the text.

In an embodiment of the computer-implemented method, a context for thetext comprises a characteristic of the text from at least one of acharacter length of the text, a character type of the text, or a phraseincluded in the other text.

In an embodiment, the computer-implemented method includes identifying aboundary of identifying information based on the context, anddetermining the range of characters based on the boundary of identifyinginformation.

In an embodiment, the computer-implemented method includes determiningthe range of characters based on a maximum, predefined range for a typeof identifying information in the text.

In an embodiment, the computer-implemented method includes replacing thesegment based on timing information of the generated text-basedrepresentation.

A computer-readable storage medium having program instructions recordedthereon that, when executed by a processing device, perform a method forscrubbing identifying information from audio and textual data, is alsodescribed. The method includes generating a text-based representation ofaudio data from a file via automatic speech recognition, the audio dataincluding data of one or more speaking parties, and determining textassociated with identifying information in the text-based representationbased on a set of key phrases and/or key words. The method also includesidentifying a portion of the text-based representation that includes apart of the text, storing modified audio data or a modified text-basedrepresentation based on removal of the identifying information. In anembodiment, the method includes replacing a segment of the audio datathat corresponds to the portion of the text-based representation withdifferent audio data that prevents determination of the identifyinginformation, and replacing the portion of the text-based representationwith different text that prevents determination of the identifyinginformation.

In an embodiment of the computer-readable storage medium, the methodincludes determining a context of at least one of the text or other textassociated with the text, and determining a range of characters toreplace the segment based on the context and the text.

In an embodiment of the computer-readable storage medium, a context forthe text comprises a characteristic of the text from at least one of acharacter length of the text, a character type of the text, or a phraseincluded in the other text.

In an embodiment of the computer-readable storage medium, the methodincludes identifying a boundary of identifying information based on thecontext, and determining the range of characters based on the boundaryof identifying information.

In an embodiment of the computer-readable storage medium, the methodincludes determining the range of characters based on a maximum,predefined range for a type of identifying information in the text.

In an embodiment of the computer-readable storage medium, the methodincludes replacing the segment based on timing information of thegenerated text-based representation.

V. CONCLUSION

While various embodiments of the disclosed subject matter have beendescribed above, it should be understood that they have been presentedby way of example only, and not limitation. It will be understood bythose skilled in the relevant art(s) that various changes in form anddetails may be made therein without departing from the spirit and scopeof the embodiments as defined in the appended claims. Accordingly, thebreadth and scope of the disclosed subject matter should not be limitedby any of the above-described exemplary embodiments, but should bedefined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A system comprising: at least one processor; anda memory that stores computer program instructions that are executableby the at least one processor, the computer program instructionsincluding: a speech recognizer configured to: receive a file thatincludes audio data of one or more speaking parties, and generate atext-based representation of the audio data via automatic speechrecognition; a context determiner configured to determine textassociated with identifying information for removal, the identifyinginformation having at least one non-numeric character therein, in thetext-based representation based on a set of key phrases; a textidentifier configured to identify a portion of the text-basedrepresentation that includes a part of the text associated with theidentifying information that has the at least one non-numeric charactertherein; and a scrubber configured to replace a segment of the audiodata that corresponds to the portion of the text-based representation,that includes the part of the text associated with the identifyinginformation that has the at least one non-numeric character therein,with different audio data.
 2. The system of claim 1, wherein thescrubber is configured to: replace the portion of the text-basedrepresentation with different text that describes a type of theidentifying information, having at least one non-numeric charactertherein, in the portion of the text-based representation.
 3. The systemof claim 1, wherein the context determiner is configured to: determine acontext of at least one of the text or other text associated with thetext; and determine a range of characters for the scrubber, to replacethe segment, based on the context and the text.
 4. The system of claim3, wherein a context for the text comprises a characteristic of the textfrom at least one of a character length of the text, a character type ofthe text, or a phrase included in the other text.
 5. The system of claim4, wherein the context determiner is configured to: identify a boundaryof identifying information based on the context; and determine the rangeof characters based on the boundary of identifying information.
 6. Thesystem of claim 4, wherein the context determiner is configured to:determine the range of characters based on a maximum, predefined rangefor a type of identifying information in the text.
 7. The system ofclaim 1, wherein the scrubber is configured to: replace the segmentbased on timing information of the generated text-based representation.8. A computer-implemented method, comprising: generating a text-basedrepresentation of audio data from a file via automatic speechrecognition, the audio data including data of one or more speakingparties; determining text associated with identifying information forremoval, the identifying information having at least one non-numericcharacter therein, in the text-based representation based on a set ofkey phrases; identifying a portion of the text-based representation thatincludes a part of the text associated with the identifying informationthat has the at least one non-numeric character therein; and replacing asegment of the audio data that corresponds to the portion of thetext-based representation, that includes the part of the text associatedwith the identifying information that has the at least one non-numericcharacter therein, with different audio data.
 9. Thecomputer-implemented method of claim 8, further comprising: replacingthe portion of the text-based representation with different text that isnot associated with the identifying information, having at least onenon-numeric character therein, and that describes a type of theidentifying information in the portion of the text-based representation.10. The computer-implemented method of claim 8, further comprising:determining a context of at least one of the text or other textassociated with the text; and determining a range of characters toreplace the segment based on the context and the text.
 11. Thecomputer-implemented method of claim 10, wherein a context for the textcomprises a characteristic of the text from at least one of a characterlength of the text, a character type of the text, or a phrase includedin the other text.
 12. The computer-implemented method of claim 11,further comprising: identifying a boundary of identifying informationbased on the context; and determining the range of characters based onthe boundary of identifying information.
 13. The computer-implementedmethod of claim 11, further comprising: determining the range ofcharacters based on a maximum, predefined range for a type ofidentifying information in the text.
 14. The computer-implemented methodof claim 8, further comprising: replacing the segment based on timinginformation of the generated text-based representation.
 15. Acomputer-readable storage medium having program instructions recordedthereon that, when executed by a processing device, perform a method forscrubbing identifying information from audio and textual data, themethod comprising: generating a text-based representation of audio datafrom a file via automatic speech recognition, the audio data includingdata of one or more speaking parties; determining text associated withidentifying information for removal, the identifying information havingat least one non-numeric character therein, in the text-basedrepresentation based on a set of key phrases; identifying a portion ofthe text-based representation that includes a part of the textassociated with the identifying information that has the at least onenon-numeric character therein; and storing modified audio data or amodified text-based representation based on removal of the identifyinginformation that has the at least one non-numeric character therein. 16.The computer-readable storage medium of claim 15, wherein the methodfurther comprises: replacing the portion of the text-basedrepresentation with different text that prevents determination of theidentifying information, having at least one non-numeric charactertherein, and that describes a type of the identifying information in theportion of the text-based representation.
 17. The computer-readablestorage medium of claim 16, wherein the method further comprisesreplacing the segment based on timing information of the generatedtext-based representation.
 18. The computer-readable storage medium ofclaim 15, wherein the method further comprises: determining a context ofat least one of the text or other text associated with the text; anddetermining a range of characters to replace the segment based on thecontext and the text.
 19. The computer-readable storage medium of claim18, wherein a context for the text comprises a characteristic of thetext from at least one of a character length of the text, a charactertype of the text, or a phrase included in the other text, and whereinthe method further comprises: identifying a boundary of identifyinginformation based on the context; and determining the range ofcharacters based on the boundary of identifying information.
 20. Thecomputer-readable storage medium of claim 18, wherein the method furthercomprises: determining the range of characters based on a maximum,predefined range for a type of identifying information in the text.